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Introduction 


Training  specific  surgical  skills  on  simulators  has  been  proven  to  bring  a  better-prepared 
student  to  a  human  operating  room,  and  when  the  simulator-trained  student  performs  a  portion  of 
a  procedure  fewer  errors  are  made  when  compared  to  a  learner  who  has  not  been  trained  on  a 
simulator.  This  current  study  seeks  to  further  this  work  by  first  developing  a  curriculum  for 
training  an  entire  procedure,  laparoscopic  cholecystectomy,  using  simulation  technologies  and 
integrating  cognitive,  psychomotor  aspects  of  full  procedure  training,  and  second,  to  test  the 
effectiveness  of  curriculum-based  training  through  a  multicenter,  international  research  group, 
the  MASTER  group. 

Body 


Thus  far  the  laparoscopic  cholecystectomy  curriculum  including  the  cognitive  and 
psychomotor  components  has  been  developed  and  validated  locally  (See  Appendix  A).  Expert 
performance  levels  have  been  established  through  three  separate  surveys  of  advanced 
laparoscopic  surgeons.  The  first  survey  was  conducted  during  the  2003  Annual  Meeting  of  the 
Society  of  Laparoscopic  Surgeons  (SLS)  where  100  surgeons  who  had  performed  in  excess  of 
100  advanced  laparoscopic  procedures  were  tested  on  the  MIST-VR.  The  second  occurred 
during  the  2003  Clinical  Congress  and  used  the  same  methodology,  and  the  third  during  the  2004 
Annual  SAGES  meeting.  Results  from  these  surveys  have  been  published  and  these  data  are 
now  part  of  a  database  for  use  in  setting  benchmarks  for  performance  on  the  MIST-VR 
(Appendix  B  &  C). 

The  methodology  for  executing  the  study  at  multiple  sites  has  been  developed  and 
distributed  (Appendix  D).  This  “cookbook”  serves  as  a  step  by  step  guide  for  setting  the  MIST- 
VR  and  establishing  local  expert  levels  for  collaborating  institutions. 

Local  IRB  approval  has  been  acquired  at  the  lead  center  and  other  sites  are  in  the  process 
of  submitting  their  IRBs. 

Several  problems  have  slowed  progress  with  this  project: 
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1 .  iRB  approval  has  been  problematic  due  to  confusion  regarding  education  research.  In 
most  cases,  education  research  is  exempt  from  full  IRB  review.  In  this  project,  the  research 
subjects  are  students,  not  patients.  When  the  student  is  in  the  operating  room  and  their 
performance  is  being  videotaped,  the  attending  surgeon  is  always  present  and  supervises  the 
student  at  all  times.  If  the  student’s  performance  of  the  operation  falls  below  what  would  be 
acceptable  for  the  patient,  the  attending  surgeon  takes  over  the  conduct  of  the  operation.  This  is 
the  standard  of  care  today  in  all  centers  with  surgical  trainees.  This  take  over  by  the  attending  is 
actually  a  data  point  which  is  recorded  when  the  videotaped  procedure  is  scored  by  blinded 
reviewers.  IRB  committees  have  found  this  methodology  confusing  thinking  that  patients  were 
being  exposed  to  untrained  surgeons  without  oversight.  Considerable  time  has  been  required  to 
educate  IRBs  on  this  experimental  design.  Within  the  last  two  months  lead  center  local  IRB 
approval  has  finally  been  secured. 

2.  The  experimental  psychologist  attached  to  this  project  resigned  his  position  at  Emory. 
In  the  6  months  preceding  this  resignation  his  work  on  this  project  declined.  Since  his 
resignation  his  duties  to  this  project  have  been  reassigned  to  another  member  of  the  research 
staff. 

3.  The  telecommunications  software  being  used  to  facilitate  the  multi-institutional 
collaboration  and  communication  for  this  project  was  acquired  by  a  different  company.  The 
company  now  managing  this  software  is  developing  it  for  different  applications  making  it 
necessary  for  us  to  look  to  a  different  vendor  to  meet  our  telecommunications  needs.  Several 
vendors  and  technologies  are  currently  being  evaluated.  A  decision  on  which  technology  to  use 
is  pending. 

4.  The  movement  of  MIS  surgeons  from  one  academic  center  to  another  has  meant  that 
some  of  the  original  collaborating  centers  no  longer  have  the  skill  and  leadership  to  participate  in 
this  research.  New  collaborators  are  being  recruited. 

Key  Research  Accomplishments 

•  Designed  curriculum  including  cognitive  and  psychomotor  components 

•  Validated  curriculum  and  presented  results  at  national  meeting  (appendix  A) 
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•  Established  expert  performance  levels  and  published  /  presented  results  (Appendix  B  & 

C) 

•  Secured  IRB  approval  at  lead  center 

•  Distributed  execution  methodology  to  collaborating  sites 


Reportable  Outcomes 

Manuscripts 

1.  Gallagher,  A.  G.,  C.  D.  Smith,  et  al.  (2003).  "Psychomotor  skills  assessment  in  practicing 
surgeons  experienced  in  performing  advanced  laparoscopic  procedures."  Journal  of  the 
American  College  of  Surgeons  197(3):  479-88. 

2.  Gallagher,  A.  G.,  A.  B.  Lederman,  et  al.  (2004).  "Discriminative  validity  of  the  Minimally 
Invasive  Surgical  Trainer  in  Virtual  Reality  (MIST-VR)  using  criteria  levels  based  on  expert 
performance."  Surgical  Endoscopy  18(4):  660-5. 

Abstracts 

1.  McClusky,  D.  M.,  C.  D.  Smith,  et  al.  (2004).  Virtual  Reality  Training  Improves  Operating 
Room  Performance  of  PGY  1  &  2  Surgical  Residents:  Results  of  a  Prospective,  Randomized, 
Double-Blinded  Study  of  the  Complete  Laparoscopic  Cholecystectomy.  Proceedings  of  the 
Surgical  Forum,  2004. 

2.  Van  Sickle,  K.  R.,  C.  D.  Smith,  et  al.  (2005).  Setting  National  Benchmark  Proficiency  Levels 
for  Laparoscopic  Performance  Using  Simulation:  The  Results  from  the  2004  SAGES  MIST- 
VR  Learning  Center  Study.  SAGES  2005  Annual  Meeting,  Accepted  for  Oral  Presentation. 

Presentations 

1.  Virtual  Reality  Training  Improves  Operating  Room  Performance  of  PGY  1  &  2  Surgical 
Residents:  Results  of  a  Prospective,  Randomized,  Double-Blinded  Study  of  the  Complete 
Laparoscopic  Cholecystectomy.  American  College  of  Surgeons  Surgical  Forum,  New  Orleans, 
LA,  Octoober  13,  2004. 
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Conclusions 


The  full  procedure  curriculum  has  been  developed  and  validated.  Lead  center  local  IRB 
approval  has  been  secured.  New  collaborating  centers  are  being  recruited.  It  is  anticipated  that 
the  24  study  subjects  will  be  enrolled  and  studies  completed  in  the  next  6-9  months.  Completion 
of  this  work  will  have  significant  impact  in  several  ways: 

1 .  This  will  be  the  first  study  to  validate  the  benefit  and  role  of  simulation  for  full 
procedure  training, 

2.  With  the  focus  on  curriculum,  and  not  just  a  specific  surgical  skill,  this  work  will 
catalyze  simulation  developers  to  progress  past  developing  technologies  that  simply  train  a 
psychomotor  skill,  but  rather,  offerings  that  incorporate  cognitive  and  psychomotor  skills  within 
a  curriculum  that  when  integrated  into  the  simulators  will  provide  a  package  more  appealing  to 
the  surgical  and  procedural  educators. 

3.  With  the  significant  logistic  issues  of  conducting  multicenter  educational  research 
resolved  through  this  project,  this  methodology  can  be  easily  reproduced  thereby  providing  a 
readily  available  mechanism  to  generate  significant  numbers  of  subjects  in  short  periods  of  time 
to  further  validate  simulation  strategies  for  industry  and  educators  alike. 


References 

1.  Gallagher,  A.  G.,  C.  D.  Smith,  et  al.  (2003).  "Psychomotor  skills  assessment  in  practicing 
surgeons  experienced  in  performing  advanced  laparoscopic  procedures."  Journal  of  the 
American  College  of  Surgeons  197(3):  479-88. 

2.  Gallagher,  A.  G.,  A.  B.  Lederman,  et  al.  (2004).  "Discriminative  validity  of  the  Minimally 
Invasive  Surgical  Trainer  in  Virtual  Reality  (MIST-VR)  using  criteria  levels  based  on  expert 
performance."  Surgical  Endoscopy  18(4):  660-5. 
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APPENDIX  A 


Presented  at  2004  ACS  Surgical  Forum 

Virtual  Reality  Training  Improves  Operating  Room  Performance  of  PGY  1  & 
2  Surgical  Residents:  Results  of  a  Prospective,  Randomized,  Double- 
Blinded  Study  of  the  Complete  Laparoscopic  Cholecystectomy. 

Objective 

To  demonstrate  that  virtual  reality  (VR)  training  improves  technical  skills  of  junior 
residents  in  the  operating  room  during  completion  of  the  full  laparoscopic 
cholecystectomy  (LC). 


Summary  Background  Data 

VR  training  has  been  demonstrated  to  improve  technical  skills  of  PGY  1-4  residents 
during  the  final  dissection  portion  of  a  LC. 


Methods 

Eleven  surgical  residents  (PGY  1  &  2)  had  baseline  psychomotor,  visio-spatial  and 
perceptual  abilities  assessed  and  were  then  randomized  to  either  Control  (n=6)  or  VR 
training  with  MIST  VR  simulator  (n=5)  until  performance  criterion  levels  established  by 
experienced  laparoscopists  were  achieved.  All  subjects  performed  a  video-recorded  LC 
with  a  supervising  attending  surgeon  blinded  to  training  status.  Video-recordings  were 
assessed  by  two  surgeon  investigators  blinded  to  subject  identity  and  training  status  and 
scored  using  pre-defined  errors  for  exposure  (n  =  11),  clipping  and  dissection  (n  =  12), 
and  dissection  of  the  gallbladder  from  the  liver-bed  (n  =  8)  with  inter-rater  reliability 
(IRR)  >0.8  (mean  IRR  =  0.96). 


Results 

VR  trained  subjects  completed  the  full  LC  20%  faster  than  controls  (31.2  v.  39.2 
minutes)  and  made  half  as  many  errors  during  exposure  of  the  cystic  duct  structures  (5.4 
v.  10,  p  <  0.04)  and  dissection  of  the  gallbladder  off  the  liver-bed  (4  v.  7.2,  p  <  0.03). 
Overall,  controls  made  twice  as  many  intra-operative  errors  (10  v.  5.4,  p  <0.04)  with  four 
times  the  variability  of  VR  trained  subjects. 

Conclusion 

Criterion  based  VR  training  for  junior  surgical  residents  transfers  to  reduced  intra¬ 
operative  errors  and  greater  performance  consistency  for  the  entire  LC. 


APPENDIX  B 


EDUCATION 


Psychomotor  Skills  Assessment  in  Practicing 
Surgeons  Experienced  in  Performing  Advanced 
Laparoscopic  Procedures 

Anthony  G  Gallagher,  PhD,  C  Daniel  Smith,  MD,  FACS,  Steven  P  Bowers,  MD,  Neal  E  Seymour,  MD,  FACS, 
Adam  Pearson,  BSc,  Steven  McNatt,  MD,  David  Hananel,  BSc,  Richard  M  Satava,  MD,  FACS 


BACKGROUND:  Minimally  invasive  surgery  (MIS)  has  introduced  a  new  and  unique  set  of  psychomotor  skills  for 
a  surgeon  to  acquire  and  master.  Although  assessment  technologies  have  been  proposed,  precise 
and  objective  psychomotor  skills  assessment  of  surgeons  performing  laparoscopic  procedures 
has  not  been  detailed. 

STUDY  DESIGN:  Two  hundred  ten  surgeons  attending  the  2001  annual  meeting  of  the  American  College  of 
Surgeons  in  New  Orleans  who  reported  having  completed  more  than  50  laparoscopic  proce¬ 
dures  participated.  Subjects  were  required  to  complete  one  box-trainer  laparoscopic  cutting  task 
and  a  similar  virtual  reality  task.  These  tasks  were  specifically  designed  to  test  only  psychomotor 
and  not  cognitive  skills.  Both  tasks  were  completed  twice.  Performance  of  tasks  was  assessed  and 
analyzed.  Demographic  and  laparoscopic  experience  data  were  also  collected. 

RESULTS:  Complete  data  were  available  on  195  surgeons.  In  this  group,  surgeons  performed  the  box- 

trainer  task  better  with  their  dominant  hand  (p  <  0.0001)  and  there  was  a  strong  and  statisti¬ 
cally  significant  correlation  between  trials  (r  =  0.47  ~  0.64,  p  <  0.0001).  After  transforming 
raw  data  to  z-scores  (mean  =  0  and  SD  =  1)  it  was  shown  that  between  2%  and  12%  of 
surgeons  performed  more  than  two  standard  deviations  from  the  mean.  Some  surgeons’  per¬ 
formance  was  20  standard  deviations  from  the  mean.  Minimally  Invasive  Surgical  Trainer 
Virtual  Reality  metrics  demonstrated  high  measurement  consistency  as  assessed  by  coefficient 
alpha  (a  =  0.849). 

CONCLUSIONS:  Objective  assessment  of  laparoscopic  psychomotor  skills  is  now  possible.  Surgeons  who  had 
performed  more  than  50  laparoscopic  procedures  showed  considerable  variability  in  their 
performance  on  a  simple  laparoscopic  and  virtual  reality  task.  Approximately  10%  of  surgeons 
tested  performed  the  task  significandy  worse  than  the  group  s  average  performance.  Studies  such 
as  this  may  form  the  methodology  for  establishing  criteria  levels  and  performance  objectives  in 
objective  assessment  of  the  technical  skills  component  of  determining  surgical  competence. 
(J  Am  Coll  Surg  2003;  197:479 — 488.  ©  2003  by  the  American  College  of  Surgeons) 


The  introduction  of  laparoscopic  or  minimally  invasive 
surgery  (MIS)  changed  the  way  patients  wanted  their 

This  study  was  supported  with  grants  from  the  American  College  of  Sur¬ 
geons,  Society  of  American  Gastrointestinal  Endoscopic  Surgeons,  Society  of 
Laparoscopic  Surgeons,  Telemedicine  and  Advanced  Technologies  Research 
Center,  and  Emory  Endosurgery  Unit,  Emory  University  School  of  Medicine. 
Mr.  Hananel  is  an  employee  of  METI,  Inc,  a  company  that  manufactures 
surgical  simulators,  though  no  METI  products  are  included  in  this  study. 

From  the  Department  of  Surgery,  University  of  Washington  Medical  Center, 
Seattle,  WA  (Gallagher,  Pearson,  Satava);  Endoscopic  Research  Laboratory, 
Belfast,  Northern  Ireland  (Gallagher);  Emory  Endosurgery  Unit,  Emory  Uni¬ 
versity  School  of  Medicine,  Atlanta,  GA  (Smith,  Bowers,  Seymour,  McNatt); 
and  Medical  Education  Technologies,  Inc,  Sarasota,  FL  (Hananel). 
Correspondence  address:  Richard  M  Satava,  MD  FACS,  University  of  Wash¬ 
ington  Medical  Center,  Department  of  Surgery,  Rm  #430,  1959  NE  Pacific 
St,  Seattle,  WA  98195. 


surgery  performed  and  the  way  surgeons  thought  about 
doing  surgery.  Although  MIS  conferred  considerable  ad¬ 
vantages  on  the  patient,  it  imposed  major  disadvantages 
on  the  surgeon.  MIS  is  more  demanding,  requiring 
greater  concentration  than  open  surgery.  Operating 
times  are  longer,  with  increased  surgical  fatigue  and 
stress  from  the  remote  interface  required  with  the  lapa¬ 
roscopic  approach.1  The  surgeon  must  reconstruct  a 
three-dimensional  mental  image  of  internal  organs  and 
structure  from  a  two-dimensional  image  provided  by  the 
charged-couple  device  (CCD)  camera  and  monitor.  Al¬ 
though  the  images  captured  by  the  CCD  device  and 
displayed  on  the  monitor  are  of  very  high  quality,  the 
images  they  produce  are  orders  of  magnitude  poorer 
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Abbreviations  and  Acronyms 

Cl  =  correct  incision 

ICI  =  incorrect  incision 

MIS  =  minimally  invasive  surgery 

MIST  VR  —  Minimally  Invasive  Surgical  Trainer  Virtual 

Reality 

VR  =  virtual  reality 


than  would  be  produced  by  the  eye  under  natural  view¬ 
ing  conditions.2  The  surgeon  also  has  to  cope  with  the 
reduced  degrees  of  freedom  of  the  surgical  instruments 
in  comparison  to  open  surgery.  This  makes  tasks  that  are 
relatively  straightforward  in  open  surgery  (eg,  suturing) 
very  difficult  with  the  minimally  invasive  approach.  Fi¬ 
nally,  the  surgeon  has  to  overcome  or  “automate”  to  the 
counterintuitive  movement  of  instruments  because  of 
the  fulcrum  effect  of  the  body  wall  on  instrument  han¬ 
dling.3  This  means  that  when  the  surgeon  moves  his 
hand  to  the  right,  the  working  end  of  the  instrument 
within  view  on  the  monitor  moves  to  the  left  and  vise 
versa.  The  fulcrum  effect  causes  a  fundamental  visual- 
proprioceptive  conflict  that  can  only  be  overcome  with 
extended  practice.4  These  problems  mean  that  mini¬ 
mally  invasive  surgeons  must  operate  at  the  very  edge  of 
their  perceptual,  cognitive,  and  psychomotor  abilities. 

When  minimally  invasive  surgery  was  first  intro¬ 
duced,  a  number  of  studies  reported  higher  complica¬ 
tions  than  open  approaches,  particularly  during  the  early 
part  of  the  minimally  invasive  surgeons  career.5,6  Other 
studies  have  reported  that  complications  persist  even 
with  very  experienced  surgeons.7,8  The  precise  reasons 
for  the  complications  experienced  by  senior  surgeons  are 
unclear.9  It  could  be  that  very  experienced  surgeons  are 
referred  more  difficult  cases,  or  there  is  a  MIS  skills 
deficit. 

Surgeons  in  training,  by  their  very  definition,  have  a 
skills  deficit,  but  these  are  often  overcome  with  training 
in  1-  to  3-day  MIS  courses,  increasing  operative  expo¬ 
sure,  and  mentoring.  But  surgeons  in  training  acquire 
these  skills  at  different  rates,  and,  indeed,  some  may  not 
ever  acquire  a  sufficient  level  of  skill  to  perform  safe 
MIS.  Cuschieri1  has  estimated  that  between  5%  and 
10%  fall  into  this  group.  One  of  the  major  problems  in 
attempting  to  establish  whether  a  surgeon  in  training  has 
acquired  the  psychomotor  skills  to  perform  MIS  is  the 
current  absence  of  benchmarks.  Rosser  and  colleagues10 
have  used  intracorporeal  suturing  as  an  indicator  of  skill 


level.  A  problem  with  the  results  of  this  assessment  of 
intracorporeal  knot  tying  is  that  time  was  the  only 
benchmark  metric.  The  inadequacy  with  this  metric  is 
that  although  a  surgeon  could  tie  a  knot  quickly,  this 
gives  no  indication  of  the  quality  of  the  knot. 

An  alternative  approach  would  be  to  use  virtual  reality 
(VR)  tasks  to  assess  performance.  The  advantage  of  this 
approach  is  that  precise  metrics  can  be  extracted  from 
MIS  performance  on  these  computer-generated  and 
tracked  tasks.  A  number  of  researchers  have  taken  this 
approach.  Several  studies  have  shown  that  a  VR  trainer 
(the  Minimally  Invasive  Surgical  Trainer  or  MIST  VR, 
Mentice  AB)  was  sensitive  enough  to  distinguish  be¬ 
tween  surgeons  of  different  levels  of  experience  in  the 
psychomotor  skills  of:  speed  of  performance  (comple¬ 
tion  time),  errors  made,  economy  of  instrument  usage 
(path  length),  and  economy  of  diathermy.11'13  As  mani¬ 
fest  by  the  designed  tasks  listed  above,  the  MIST  VR 
simulator  is  designed  to  test  only  perceptual  abilities  and 
psychomotor  skills;  there  is  no  component  of  cognitive 
skills  or  decision  making  in  the  simulation.  In  a  later 
study,  Gallagher  and  Satava14  investigated  these  designed 
psychometric  properties  of  MIST  VR.  They  found  that 
the  simulator  had  a  high  test-retest  reliability  and  a  high 
alpha  coefficient  (ie,  a  measure  of  internal  measurement 
consistency),  and  distinguished  between  surgeons  in 
terms  of  learning  curves  and  variability  of  performance 
(construct  validity).  All  of  these  studies  concluded  that 
MIST  VR  could  be  a  useful  device  for  assessing  MIS 
performance  in  the  laboratory,  particularly  because  of  its 
performance  metrics  for  psychomotor  skills. 

Following  the  above  validation  (face,  concurrent, 
construct,  and  content  validity)  of  the  MIST  VR  as  a 
system  to  train  and  assess  psychomotor  skills,  the  simu¬ 
lator  was  evaluated  for  predictive  validity:  Is  the  system  a 
valid  predictor  of  the  performance  of  psychomotor  (not 
cognitive)  skills  in  the  operating  room?  Seymour  and 
colleagues15  demonstrated  that  residents  trained  on  the 
MIST  VR  simulator  (as  compared  with  a  control  group 
with  no  simulator  training)  made  fewer  errors  and  used 
less  time  for  the  gallbladder  excision  portion  of  a  lapa¬ 
roscopic  cholecystectomy  on  patients. 

But  a  major  obstacle  to  implementing  MIST  VR  as  an 
assessment  device  is  that  the  studies  that  have  been  con¬ 
ducted  have  used  only  small  numbers  of  participants.  It 
is  also  not  clear  how  an  individuals  performance  on  a 
VR  task  relates  to  performance  of  an  entire  surgical  pro¬ 
cedure  on  a  real-world  task.  More  important,  there  is  no 
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study  to  date  (and  it  may  take  decades)  to  prove  that 
training  on  a  simulator  directly  improves  outcomes  (pa¬ 
tient  safety,  decreased  complications,  higher  quality  of 
life). 

The  purpose  of  the  study  reported  here  was  to  bench¬ 
mark  MIS  performance  of  psychomotor  skills  of  experi¬ 
enced  minimally  invasive  surgeons  on  a  VR  task  and  a 
similar  box-trainer  task.  The  box-trainer  task,  a  simple 
laparoscopic  paper-cutting  task,  was  chosen  because  it 
was  first  used  to  empirically  demonstrate  the  impact  of 
the  fulcrum  effect  on  MIS  performance  and  has  subse¬ 
quently  been  used  extensively  to  assess  different  MIS 
training  programs.3,16'19  It  has  also  been  demonstrated  to 
have  low  variability,  normally  distributed  performance, 
and  is  sensitive  to  learning  and  errors.  This  relatively 
simple  laparoscopic  task  also  requires  little  cognitive  ef¬ 
fort  (ie,  remembering  what  to  do),  so  is  a  good  measure 
of  laparoscopic  psychomotor  performance.  For  the  pur¬ 
poses  of  this  study  a  similar  VR  task  was  constructed.  It 
was  predicted  that  experienced  minimally  invasive  sur¬ 
geons  would  demonstrate  variability  in  their  perfor¬ 
mance  on  both  a  virtual  reality  task  and  a  box-trainer 
task,  although  the  majority  of  surgeons’  performance 
would  fall  within  a  range  plus  or  minus  two  standard 
deviations  from  the  mean.  Another  prediction  was  that 
performance  on  the  box-trainer  and  VR  task  would  cor¬ 
relate  strongly  enough  to  allow  only  the  VR  task  to  be 
used  in  future  studies. 

METHODS 

Subjects 

Surgeons  attending  the  2001  annual  meeting  of  the 
American  College  of  Surgeons  in  New  Orleans  parti¬ 
cipated.  Two  hundred  ten  surgeons  were  recruited,  and 
15  surgeons  failed  to  complete  any  single  task.  Their 
data  were  excluded  from  subsequent  analysis.  Demo¬ 
graphic  details  of  surgeons  who  participated  are  shown 
in  Table  1. 

Apparatus 
MIST  VR 

The  MIST  VR  system  comprises  a  standard  200-MHz 
PC  with  32  Mb  RAM,  linked  to  a  jig  containing  two 
laparoscopic  instruments  held  in  position-sensing 
gimbles  with  5  degrees  of  freedom.  This  provides  real¬ 
time  translation  of  the  instrument  movements  to  the 
graphic  display  on  a  15-inch  color  monitor.  An  accu¬ 
rately  scaled  operating  volume  of  10  cm3  is  represented 


Table  1.  Demographic  Details  of  Surgeons  Tested. 


Age  (y) 

30-39 

24.6% 

40-49 

40% 

50-59 

23% 

60-69 

12.4% 

>70 

0% 

Mean  number  of  career 
laparoscopic  cases 

967  (SD  =  994;  range  20-4,000) 

Mean  number  of  laparoscopic 

144  (SD  =  105;  range  2-400) 

cases  per  year 

Right  hand  dominant 

92.3% 

Male 

86% 

by  a  three-dimensional  cube  on  the  computer  screen. 
The  image  zoom  and  size  of  the  target  objects  can  be 
varied.  Circular  targets  (at  a  diameter  of  12  mm)  appear 
randomly  within  the  operating  volume  and  can  be 
“grasped”  and  “manipulated.”  The  MIST  VR  trainer 
recorded  time,  error,  and  economy  of  instrument  move¬ 
ment  relative  to  the  target  object.  A  novel  MIST  VR  task 
was  developed  for  this  study  and  is  shown  in  Figure  1 .  It 
consisted  of  a  rectangular  virtual  card  with  five  spheres 
along  the  long  edge  and  a  larger  sphere  in  the  middle  of 
the  short  edge.  The  large  sphere  indicated  which  hand  or 
instrument  was  to  be  used  to  grasp  the  card  and  where 
subjects  should  start  excising  the  spheres.  The  hand  or 
instrument  used  to  grasp  the  card  alternated  with  each 
card  completed. 


Figure  1.  The  new  MIST  VR  task  for  the  2001  American  College  of 
Surgeons  objective  assessment  of  laparoscopic  psychomotor  skill 
program.  MIST  VR,  Minimally  Invasive  Surgical  Trainer  Virtual  Real¬ 
ity. 
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Figure  2.  The  portable  laparoscopic  box  trainer. 


Box-trainer  task 

A  standard  laparoscopic  cutting  task  was  used  as  previ¬ 
ously  reported.3,16  Briefly,  a  cut  is  made  in  26  spaces 
clearly  demarcated  by  black  lines  along  the  long  edge  of 
a  sheet  of  paper.  The  task  was  performed  in  a  Portable 
Laparoscopic  Trainer  (3-D  Technical  Services,  Franklin, 
OH)  (Fig.  2).  Ethicon  nontoothed  laparoscopic  forceps 
(Ethicon  Endosurgery,  Cincinnati,  OH)  and  laparo¬ 
scopic  curved  scissors  were  used  to  hold  and  cut  the 
paper.  This  task  has  been  shown  to  be  sensitive  to  differ¬ 
ences  between  experienced  and  junior  surgeons,  is  sen¬ 
sitive  to  learning,  measures  correct  responses  and  errors, 
has  low  variability,  and  has  been  demonstrated  to  con¬ 
form  to  the  Gaussian  distribution. 

Procedure 

Subjects  reported  to  the  testing  booth  in  the  scientific 
exhibition  area  of  the  conference  and  completed  a  ques¬ 
tionnaire  detailing  their  experience  and  training.  After 
confirming  that  they  had  completed  more  than  50  lapa¬ 


roscopic  procedures,  they  were  given  a  unique  identifier 
number  and  then  watched  a  video  recording  explaining 
how  to  perform  the  laparoscopic  tasks,  including  what 
constituted  an  error.  Subjects  rotated  through  four  test¬ 
ing  stations  in  alternate  order,  two  virtual  reality  stations 
and  two  box-trainer  cutting  stations.  The  virtual  reality 
task  required  subjects  to  hold  the  virtual  card  with  one 
instrument  and  excise  the  spheres  by  grasping  them  in 
the  center  with  the  other  instrument.  When  this  was 
done  correctly  the  sphere  disappeared.  The  virtual  real¬ 
ity  task  always  started  with  the  subjects  holding  the 
sphere  with  their  nondominant  hand  and  excising  the 
spheres  with  their  dominant  hand.  After  all  five  spheres 
had  been  excised,  a  new  card  appeared  and  was  to  be 
grasped  with  the  dominant  hand.  All  subjects  were  re¬ 
quired  to  complete  four  cards  with  a  total  of  20  spheres. 
Timing  was  stopped  between  all  the  spheres  being  ex¬ 
cised  for  one  card  and  a  new  card  appearing.  MIST  VR 
recorded  time,  errors,  and  economy  of  instrument 
movement. 

The  laparoscopic  cutting  task  was  placed  horizontally 
in  a  conventional  box  trainer  under  standardized  lapa¬ 
roscopic  conditions.  Subjects  were  required  to  make  one 
incision  between  26  spaces  clearly  demarcated  by  black 
lines  along  the  long  edge  of  the  sheet  of  paper  (US  let¬ 
ter).  Subjects  grasped  the  paper  with  their  nondominant 
hand  and  cut  with  their  dominant  hand  for  the  first  trial 
and  then  swapped  over  for  the  second  trial.  An  error  was 
judged  to  have  been  made  if  a  subjects  incision  cut 
across  one  of  the  black  lines  or  touched  it.  If  an  incision 
was  so  close  to  the  line  that  the  experimenter  found  it 
difficult  to  make  a  decision  it  was  judged  an  error. 
Twenty  percent  of  sheets  were  rechecked  by  another  in¬ 
vestigator.  The  inter-rater  reliability  was  98%.  Subjects 
were  asked  to  make  as  many  incisions  as  they  could  in 
two  2-minute  periods.  The  laparoscopic  task  was  re¬ 
moved  from  the  box  trainer  after  each  subject  was  fin¬ 
ished.  Timing  commenced  when  subjects  placed  the  pa¬ 
per  between  the  jaws  of  the  scissors  for  their  first  incision. 
Subjects  were  stopped  after  60  seconds  had  elapsed. 

RESULTS 

The  means  and  standard  deviations  of  surgeon  perfor¬ 
mance  on  the  box  trainer  and  VR  tasks  are  shown  in 
Table  2.  Differences  between  performance  on  the  two 
trials  were  compared  for  significance  with  ANOVA  for 
repeated  measures  and  these  results  are  shown  in  Table  3. 

Results  for  the  box-trainer  task  reveal  that  surgeons 
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Table  2.  Mean  Scores  by  Surgeons  on  the  Box  Trainer  and  Virtual  Reality  Tasks  for  Trials  1  and  2  and  Their  Intercorrelation 


Trial  1 
mean  (SD) 

Trial  2 
mean  (SD) 

r 

df 

F  value 

p  Value 

Box  trainer  metrics 

Correct  incisions 

19.1  (5.4) 

17.7  (6) 

1.191 

135.1 

0.0001 

Incorrect  incisions 

3.03  (7.3) 

5.14(13) 

0.472 

1.191 

54.6 

0.0001 

MIST  VR  metrics 

Time 

98.9  (54.3) 

76.51  (43) 

0.487 

1.183 

56.3 

0.0001 

Errors 

64.3  (54.4) 

53  (38.5) 

0.48 

1.181 

9.59 

0.0023 

Economy  of  movement 

15.7(8.5) 

12.75  (6.2) 

0.553 

1.182 

79.7 

0.0001 

MIST  VR,  Minimally  Invasive  Surgical  Trainer  Virtual  Reality. 


made  significantly  more  correct  incisions  (Cl)  with  their 
dominant  hand  in  trial  1  than  they  did  with  their  non¬ 
dominant  hand  during  trial  2.  They  also  made  signifi¬ 
cantly  more  incorrect  incisions  (ICI)  during  trial  2.  The 
percentage  of  ICI  scores  showed  considerable  variability, 
with  standard  deviations  that  were  twice  as  large  as  the 
mean  scores  for  both  trials. 

Performance  as  a  function  of  hand  dominance  was 
not  assessed  separately  for  MIST  VR  because  the  tasks 
alternated  which  hand  did  most  of  the  work.  Across  all 
three  measures  there  was  a  statistically  significant  im¬ 
provement  between  trials  1  and  2.  There  was  consider¬ 
able  variability  in  all  of  the  VR  scores  on  trial  1  but  this 
had  decreased  by  trial  2.  Pearsons  Product  Moment 
Correlation  Coefficient  was  used  to  assess  how  strongly 
performance  on  the  first  trial  was  related  to  performance 
on  the  second  trial.  These  are  presented  in  Table  1  with 
F  values  and  probability  levels.  Performance  on  trial  1 
was  strongly  and  statistically  significantly  related  to  per¬ 
formance  on  trial  2  across  both  the  box  trainer  and 
MIST  VR  measures,  with  correlations  ranging  from  r  = 
0.48  to  0.644. 

For  ease  of  analysis  all  scores  for  each  measure  were 
transformed  to  z-scores  using  the  mean  and  standard 
deviation  of  each  measure  for  each  trial.  The  advantage 
of  this  approach  is  that  the  mean  =  0  with  a  standard 
deviation  =  1.  Any  scores  falling  more  than  two  stan¬ 
dard  deviations  either  way  from  the  mean  differ  statisti¬ 
cally  significantly  at  the  95%  probability  level  from  the 
sample  mean.  Figure  3A  shows  the  distribution  of  Cl 
z-scores  for  trials  1  and  2  (dominant  hand  on  trial  1  and 
nondominant  hand  on  trial  2).  From  the  graph  it  can  be 
seen  that  some  individuals  are  scoring  four  standard  de¬ 
viations  from  the  mean,  indicating  that  they  are  making 
dramatically  fewer  Cl  than  the  mean  performance: 
2.1%  in  trial  1  and  2.6%  of  subjects  in  trial  2.  A  statis¬ 
tical  problem  created  by  these  oudiers  is  that  they  were 


used  in  the  calculation  of  the  z-scores,  so  will  likely  neg¬ 
atively  bias  the  distribution.  To  eliminate  this  problem, 
new  means  and  standard  deviations  were  calculated  for 
each  of  the  metrics  and  for  each  trial  based  only  on  the 
scores  of  surgeons  who  fell  within  plus  or  minus  two 
standard  deviations.  These  are  presented  in  Table  4. 

All  subsequent  graphs  show  the  raw  z-scores  on  the 
upper  plate  (A)  and  the  biweighted  z-scores  on  the  lower 
plate  (B).  Even  with  the  biweighted  transformation 
shown  in  Figure  3B,  4.1%  of  surgeons  scored  more  than 
two  standard  deviations  away  from  the  mean. 

Figures  4A  and  4B  show  the  distribution  of  scores  for 
ICIs.  Most  surgeons’  performance  bunched  around  the 
mean,  but  both  distributions  exhibit  a  long  tail.  For  the 
raw  z-scores,  5.64%  of  surgeons  scored  more  than  two 
standard  deviations  away  from  the  mean,  but  for  the 
biweighted  mean  this  was  12.3%.  Indeed,  for  the  bi¬ 
weighted  distribution  some  surgeons  scored  more  than 
13  standard  deviations  from  the  mean  on  trial  1  and  up 
to  20  on  trial  2. 

Although  the  MIST  VR  time  scores  show  a  more  even 
distribution  (Figs.  5A  and  5B)  than  the  ICI  scores,  they 
also  exhibit  a  long  tail.  For  the  raw  z-scores,  6.3%  and 
7%  of  surgeons  scored  more  than  two  standard  devia- 


Table  3.  Results  of  ANOVA  for  Repeated  Measure  Between 
the  Difference  in  Performance  on  Trial  1  and  Trial  2 


df 

F  value 

p  Value 

Box  trainer 

metrics 

Correct  incisions 

1,191 

16.45 

0.0001 

Incorrect  incisions 

1,191 

54.6 

0.0001 

MIST  VR  metrics 

Time 

1,183 

36.52 

0.0001 

Errors 

1,181 

54.3 

0.0023 

Economy  of 
movement 

1,182 

30.76 

0.0001 

MIST  VR,  Minimally  Invasive  Surgical  Trainer  Virtual  Reality. 
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B 

Figure  3.  (A  and  B)  Distribution  of  z-scores  for  correct  incisions 
made  during  the  two  2-minute  trials. 

tions  from  the  mean  for  trials  1  and  2,  respectively.  The 
biweighted  z-distribution  exhibited  11.6%  and  8.6%. 
No  surgeon  performed  more  than  five  standard  devia¬ 
tions  from  the  mean. 

MIST  VR  error  scores  are  presented  in  Figures  6A  and 
6B.  Most  surgeon  performance  bunches  around  the 
mean,  but  both  distributions  exhibit  a  long  tail.  In  trials 
1  and  2,  5.8%  and  5.4%  scored  more  than  two  standard 


Table  4.  Biweighted  Mean  Scores  by  Surgeons  on  the  Box 
Trainer  and  Virtual  Reality  Tasks  for  Trials  1  and  2 


Trial  1 
mean  (SD) 

Trial  2 
mean  (SD) 

Box  trainer  metrics 

Correct  incisions 

19.4  (5.03) 

18.05  (5.89) 

Incorrect  incisions 

1.37  (2.95) 

3.02  (5.28) 

MIST  VR  metrics 

Time 

89.7  (39.49) 

68.22  (27.28) 

Errors 

55.23  (38.5) 

47.72  (28.61) 

Economy  of  movement 

14.38(6.12) 

11.78(4.3) 

MIST  VR,  Minimally  Invasive  Surgical  Trainer  Virtual  Reality. 


deviations  away  from  the  mean  for  the  raw  z-scores, 
respectively.  This  increased  to  10%  and  10.7%  for  the 
biweighted  distribution.  Indeed,  some  surgeon  perfor- 


Flgure  4.  (A  and  B)  Distribution  of  z-scores  for  incorrect  incisions 
made  during  the  two  2-minute  trials. 
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Figure  5.  (A  and  B)  Distribution  of  z-scores  for  time  performance 
metric  on  MIST  VR  for  trials  1  and  2.  MIST  VR,  Minimally  Invasive 
Surgical  Trainer  Virtual  Reality. 

mance  fell  seven  standard  deviations  from  the  mean  for 
both  trials  1  and  2. 

Economy  of  instrument  movement  exhibited  a  simi¬ 
lar  pattern  to  the  error  scores.  Figures  7A  and  7B  show 
that  most  of  the  scores  bunch  around  the  mean,  and, 
similar  to  the  other  scores,  exhibit  a  long  tail  The  top 
plate  (Fig.  7A)  shows  that  5.8%  and  5.3%  of  surgeons  in 
trials  1  and  2  performed  more  than  two  standard  devia¬ 
tions  from  the  mean.  For  the  biweighted  distribution  it 
was  11.5%  and  9.6%,  respectively.  One  surgeon  per¬ 
formed  eight  standard  deviations  from  the  mean.  Coef¬ 
ficient  alpha  was  used  to  assess  the  internal  measurement 
consistency  of  the  MIST  VR  metrics  and  was  observed 
to  be  high;  a  =  0.849. 
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Figure  6.  (A  and  B)  Distribution  of  z-scores  for  error  performance 
metric  on  MIST  VR  for  trials  1  and  2.  MIST  VR,  Minimally  Invasive 
Surgical  Trainer  Virtual  Reality. 

DISCUSSION 

Most  studies  on  the  objective  assessment  of  laparoscopic 
performance  have  only  used  small  numbers  of  subjects 
(eg,  9,  10,  11,  12)  and  when  larger  numbers  have  been 
used  only  crude  measures  (eg,  time)  have  been  reported. 
This  is  one  of  the  first  studies  to  use  both  a  box-trainer 
task  and  a  virtual  reality  task.  We  also  tested  only  expe¬ 
rienced  laparoscopic  surgeons,  ie,  surgeons  who  claimed 
to  have  performed  more  than  50  laparoscopic  opera¬ 
tions.  Surgeons  performed  better  with  their  dominant 
hand  in  trial  1  of  the  box-trainer  cutting  task.  On  the 
MIST  VR  tasks  there  was  a  significant  improvement 
between  trials  1  and  2. 

One  goal  of  this  study  was  to  objectively  assess 
whether  some  surgeons  were  performing  worse  than  the 
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Figure  7.  (A  and  B)  Distribution  of  z-scores  for  economy  of  move¬ 
ment  performance  metric  on  MIST  VR  for  trials  1  and  2.  MIST  VR, 
Minimally  Invasive  Surgical  Trainer  Virtual  Reality. 

mean  and,  if  so,  how  much  worse.  The  box-trainer  and 
VR  task  were  chosen  because  they  were  extremely  simple 
and  involved  little  cognition  or  decision  making,  so  they 
should  have  given  a  pure  measure  of  psychometric  per¬ 
formance.  Indeed,  before  the  data  collection  started, 
some  of  the  coauthors  expressed  concern  that  the  tasks 
were  too  simple  and  would  not  be  challenging  enough 
for  experienced  laparoscopic  surgeons.  After  data  collec¬ 
tion  started  it  was  evident  that  even  such  a  simple  basic 
skill  was  a  sensitive  discriminator.  Some  surgeons  ex¬ 
celled  at  both  tasks,  but  others  had  great  difficulty  per¬ 
forming  either  the  VR  or  the  box-trainer  task.  The  re¬ 
sults  we  have  presented  here  are  from  only  surgeons  who 
completed  some  part  of  the  study.  Fifteen  surgeons  had 


to  be  excluded  because  they  did  not  complete  any  part  of 
the  study. 

One  of  the  findings  that  surprised  all  the  authors  was 
how  poorly  some  surgeons  performed  (eg,  up  to  20  stan¬ 
dard  deviations  from  the  mean).  Reassuringly,  this  num¬ 
ber  was  extremely  small.  But  what  this  study  has  dem¬ 
onstrated  is  that  based  on  objective  metrics  it  is  possible 
to  identify  individuals  who  have  laparoscopic  basic  psy¬ 
chomotor  skill  deficits.  Both  the  box-trainer  task  and  the 
VR  task  achieved  this  and  corroborate  the  veracity  of  the 
results.  But  the  VR  task  generates  the  metrics  automat¬ 
ically,  and  records  performance  and  provides  more  com¬ 
prehensive  performance  metrics  in  comparison  to  the 
box-trainer  task,  in  which  the  experimenter  has  to  phys¬ 
ically  count  the  number  of  incisions.  Although  the  inter¬ 
rater  reliability  observed  in  this  study  was  very  high, 
there  is  always  the  possibility  of  human  error  with  the 
box  trainer.  The  box-trainer  task  that  was  used  was  sim¬ 
ple,  with  only  two  possible  outcomes,  both  of  which 
were  clearly  defined.  Had  a  more  complicated  task  been 
used,  the  inter-rater  reliability  may  not  have  been  as 
high.  MIST  VR  has  also  been  the  subject  of  extensive 
validation  studies9'12  and  is  currently  the  best  validated 
VR  system  in  surgical  education.  These  MIST  VR  vali¬ 
dation  studies  used  the  original  six  tasks  that  were  devel¬ 
oped  to  teach  psychomotor  skills  and  instrument  han¬ 
dling  for  laparoscopic  cholecystectomy.  The  study 
reported  here  used  a  novel  task  to  eliminate  the  effect  of 
earlier  exposure.  No  surgeon  had  worked  on  the  new  VR 
task  before  the  American  College  of  Surgeons  2001 
meeting.  One  advantage  that  the  box-trainer  task  had 
over  the  VR  task  was  that  dominant  and  nondominant 
hand  performance  were  assessed  separately.  In  a  new 
study  we  plan  to  assess  dominant  hand  performance  on 
both  the  box  trainer  and  virtual  environments  and  how 
strongly  performance  on  one  correlates  with  the  other. 

Currently,  surgical  residents  wishing  to  pursue  a  ca¬ 
reer  in  minimally  invasive  surgery  have  no  objective  na¬ 
tional  or  international  benchmarks  at  which  to  aim. 
They  can  only  use  local  attending  surgeon  performance, 
and  frequently  the  local  training  institution  has  not  val¬ 
idated  a  criterion  level  to  achieve,  so  training  is  focused 
on  training  time,  rather  than  objective  criteria  for  profi¬ 
ciency.  What  the  data  from  this  study  have  shown  is  that 
there  is  considerable  variability  in  surgeon  performance, 
which  we  assume  translates  nationally.  We  believe  that 
this  issue  will  become  more  important.  In  the  recent 
study  by  Seymour  and  Colleagues,15  the  benefits  of 
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benchmarking  were  clear.  In  a  prospective,  randomized, 
double-blind  trial  of  VR  training  versus  standard  surgi¬ 
cal  training,  the  researchers  found  that  surgical  residents 
trained  on  VR  simulators  made  significandy  fewer  errors 
during  a  laparoscopic  cholecystectomy.  One  of  the  cru¬ 
cial  aspects  of  this  study  was  that  the  residents  were 
trained  to  an  objectively  measured  score  (as  determined 
by  assessment  of  attending  surgeon  psychomotor  perfor¬ 
mance  levels  on  the  VR  simulator),  and  were  required  to 
reach  this  established  performance  level  (criteria)  on  two 
consecutive  trials  before  being  allowed  to  operate  on  a 
patient  ,15  The  results  of  that  study  suggested  that  surgi¬ 
cal  residents  clearly  demonstrated  their  ability  to  achieve 
a  criterion  level,  but  some  took  longer  than  others.  Sur¬ 
gical  residents  should  be  given  clear  guidance  as  to  the 
standard  of  psychomotor  performance  they  should 
achieve  (benchmark)  before  operating  on  a  patient. 

Study  limitations 

One  of  the  limitations  of  this  study  is  that  the  “experts” 
volunteered,  so  they  self-defined  “laparoscopic  sur¬ 
geon.”  We  have  no  objective  independent  information 
on  what  types  of  MIS  they  performed  other  than  their 
self-reports,  and  we  have  no  information  on  their  per¬ 
formance  in  the  operating  room,  ie,  outcomes  data.  An¬ 
other  limitation  of  this  study  was  that  the  tasks  were 
probably  too  easy  for  the  expert  laparoscopic  surgeon. 
This  was  most  clearly  demonstrated  on  the  box-trainer 
task,  where  some  surgeons  made  incisions  in  all  26 
spaces,  with  no  errors,  in  less  than  a  minute.  Other  fac¬ 
tors  that  might  have  affected  performance  relate  to  the 
fact  that  it  was  in  a  busy  booth  in  the  convention  exhibit 
hall;  there  were  distractions  from  the  convention  or  they 
may  have  been  tired  after  a  long  flight  or  busy  meeting 
schedule.  Last,  because  of  time  constraints,  demo¬ 
graphic  information  was  self-reported  rather  than  objec¬ 
tively  assessed;  eg,  handedness. 

In  conclusion,  this  study  has  shown  that  using  already 
validated  methods  of  laparoscopic  skills  assessment,  it  is 
possible  to  measure  laparoscopic  psychomotor  perfor¬ 
mance  of  laparoscopic  surgeons  who  had  performed 
more  than  50  laparoscopic  procedures  by  using  a  simple 
box-trainer  task  and  a  virtual  reality  task.  Between  2% 
and  12%  of  surgeons  assessed  fell  more  than  two  stan¬ 
dard  deviations  away  from  the  mean.  The  majority  of 
surgeons  fell  within  the  two  standard  deviations  but  con¬ 
siderable  variability  in  performance  was  observed.  Some 
surgeons’  scores  fell  20  standard  deviations  from  the 


mean.  This  type  of  performance  is  unlikely  to  have  oc¬ 
curred  by  chance  because  a  large  and  statistically  signif¬ 
icant  correlation  was  observed  between  trials. 

Author  contributions 

Study  conception  and  design:  Gallagher,  Smith,  Bowers, 
Seymour,  Pearson,  McNatt,  Hananel,  Satava 
Acquisition  of  data:  Gallagher,  Smith,  Bowers,  Seymour, 
Pearson,  McNatt,  Hananel 
Drafting  of  manuscript:  Gallagher 
Critical  revision:  Gallagher,  Smith,  Seymour,  Satava 
Statistical  expertise:  Gallagher 
Obtaining  funding:  Smith,  Hananel,  Satava 

REFERENCES 

1.  Cuschieri  A.  Whither  minimal  access  surgery:  Tribulations  and 
expectations.  Am  J  Surg  1995;169:9-19. 

2.  Cuschieri  A.  Visual  displays  and  visual  perception  in  Minimal 
Access  Surgery.  Sem  Lap  Surg  1995;2:209-214. 

3.  Reinhardt-Rutland,  AH,  Gallagher  AG.  Visual  depth  percep¬ 
tion  in  minimally  invasive  surgery.  In:  Robertson  SA,  ed.  Con¬ 
temporary  ergonomics.  London:  Taylor  &  Francis;  1996:531— 
536. 

4.  Gallagher  AG,  McClure  N,  McGuigan  J,  et  al.  An  ergonomic 
analysis  of  the  ‘Fulcrum  Effect’  in  endoscopic  skill  acquisition. 
Endoscopy  1998;30:617-620. 

5.  Crothers  I,  Gallagher  AG,  McClure  N,  et  al.  Experienced  sur¬ 
geons  are  automated  to  the  fulcrum  effect;  an  ergonomic  dem¬ 
onstration.  Endoscopy  1999;31:365-369. 

6.  Wherry  DC,  Rob  CG,  Marohn  MR,  et  al.  An  external  audit  of 
laparoscopic  cholecystectomy  performed  in  medical  treatment 
facilities  of  the  Department  of  Defense.  Ann  Surg  1994;220: 
626-634. 

7.  The  Southern  Surgeons  Club.  The  learning  curve  for  laparo¬ 
scopic  cholecystectomy.  Am  J  Surg  1995;170:55-59. 

8.  Gigot  J-F,  Etienne  J,  Aerts  R,  et  al.  The  dramatic  reality  of 
biliary  tract  injury  during  laparoscopic  cholecystectomy;  an 
anonymous  multi-center  survey  of  65  patients.  Surg  Endosc 
1997;11:1171-1178. 

9.  Archer  SB,  Brown  DB,  Smith  CD,  et  al.  Bile  duct  injury  during 
laparoscopic  cholecystectomy:  Results  of  a  national  survey.  Ann 
Surg  2001;234:549-559. 

10.  Rosser  JC,  Rosser  LE,  Savalgi  RS.  Objective  evaluation  of  lapa¬ 
roscopic  surgical  skill  program  for  residents  and  senior  surgeons. 
Arch  Surg  1998;133:657-661. 

11.  Smith  CD,  Farrell  TM,  McNatt  SS,  Metreveli  RE.  Assessing 
laparoscopic  manipulative  skills.  Am  J  Surg  200 1 ;  1 8 1 :547-550. 

12.  Taffinder  N,  Sutton  C,  Fishwick  RJ,  et  al.  Validation  of  victual 
reality  to  teach  and  assess  psychomotor  skills  in  laparoscopic 
surgery:  results  from  randomised  controlled  studies  using  the 
MIST  VR  laparoscopic  trainer.  Stud  Health  Tech  Inform  1998; 
50:124-130. 

13.  McNatt  SS,  Smith  CD.  A  computer-based  laparoscopic  skills 
assessment  device  differentiated  experience  from  novice  laparo¬ 
scopic  surgeons.  Surg  Endosc  2001;15:1085-1089. 

14.  Gallagher  AG,  McGuigan  J,  Ritchie  K,  et  al.  Objective  psy- 


488 


Gallagher  et  al 


Psychomotor  Skills  Assessment 


J  Am  Coil  Surg 


chomotor  assessment  of  senior,  junior  and  novice  laparoscopists 
with  virtual  reality.  World  J  Surg  2001;25:1478-1483. 

15.  Seymour  N,  Gallagher  AG,  Satava  RM,  O’Brien,  et  al.  Virtual 
reality  training  improves  operating  room  performance:  Results 
of  a  randomized,  double-blinded  study.  Ann  Surg  2002;238: 
458-464. 

16.  Gallagher  AG,  Satava  RM.  Objective  assessment  of  experience, 
junior  and  novice  laparoscoic  performance  with  virtual  reality: 
Learning  curves  and  reliability  measures.  Surgl  Endosc  (In 
press) . 

17.  Gallagher  AG,  Hughs  C,  Reinhardt-Rutland  AH,  et  al.  Transfer 


of  skill  from  Virtual  reality’  (VR);  a  case  control  comparison  of 
traditional  and  VR  training  in  laparoscopic  skill  acquisition. 
MITAT  2000;9:347-352. 

18.  Jordan  JA,  Gallagher  AG,  McGuigan  J,  et  al.  A  comparison 
between  randomly  alternating  imaging,  normal  laparoscopic 
imaging  and  virtual  reality  training  in  laparoscopic  psychomotor 
skill  acquisition.  Am  J  Surg  2000;180:208-21 1. 

19.  Jordan  JA,  Gallagher  AG,  McGuigan  J,  et  al.  Virtual  reality 
training  leads  to  faster  automation  of  the  novel  psychomotor 
restrictions  encountered  by  laparoscopic  surgeons.  Surg  Endosc 
2001;15:1080-1084. 


APPENDIX  C 


Accepted  for  Oral  Presentation  at  SAGES  2005  Annual  Meeting 
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Background:  The  Minimally  Invasive  Surgical  Trainer  Virtual  Reality  (MIST-VR)  (Mentice, 
Gottenberg,  Sweden)  has  been  well  validated  as  a  training  device  for  laparoscopic  skills. 
Training  to  a  level  of  proficiency  on  the  simulator  has  been  demonstrated  to  significantly  improve 
objectively  assessed  operating  room  performance  during  laparoscopic  cholecystectomy.  The 
purpose  of  this  project  was  to  establish  a  national  standard  of  proficiency  on  the  simulator  based 
on  the  performance  of  experienced  laparoscopic  surgeons.  Methods:  Surgeons  attending  the 
SAGES  2004  Annual  Meeting  who  had  performed  more  than  100  laparoscopic  procedures 
volunteered  to  participate  and  were  tested  in  the  SAGES  Learning  Center.  All  subjects 
completed  a  demographic  questionnaire  to  assess  laparoscopic  and/or  MIST-VR  experience. 
Each  subject  performed  two  consecutive  trials  of  the  MIST-VR  Core  Skills  1  program  on  medium 
settings  (six  basic  tasks  of  increasing  difficulty;  acquire  place  (AP),  transfer  place  (TP),  traversal 
(TV),  withdrawal  insert  (Wl),  diathermy  task  (DT),  manipulate  diathermy  (MD)).  Trial  1  was 
considered  a  “warm-up”  and  Trial  2  functioned  as  the  test  trial  proper.  Subject  performance  was 
scored  for  time,  errors  and  economy  of  instrument  movement  for  each  task,  and  a  cumulative 
total  score  was  calculated.  Results:  57  surgeons  participated  in  the  study,  complete  data  is 
available  for  42.  Trial  2  data  expressed  as  mean±SD;  time  in  seconds;  other  values  unitless. 


Task 

Econ  L 

Errors  L 

Errors  R 

Tot.  Time 

Score 

AP 

2.6±0.6 

5.0±4.3 

4.1±3.2 

12.512.8 

26.719.2 

TP 

3.0±1.0 

2. 8+0.8 

8.8±7.0 

7.8±6.0 

18.415.1 

38.4114.0 

TV 

5.3±2.0 

5.1  ±1.8 

15.7±14.0 

11.2110.3 

36.6114.5 

73.8134.9 

Wl 

2.0±0.3 

2.0±0.5 

4.7±4.1 

4.415.7 

15.314.3 

27.3111.4 

DT 

2.7±0.7 

7.7±5.1 

20.514.7 

41.6111.4 

MD 

2.410.6 

2.3±0.7 

51.9±35.3 

53.3134.1 

50.9126.9 

160.7192.9 

MIST-VR  Core  Skills  1  Total  Score 

374.81134 

Conclusion:  National  benchmark  proficiency  levels  for  laparoscopic  skills  have  now  been 
established  by  experienced  laparoscopic  surgeons  using  the  MIST-VR  simulator.  Residency 
programs,  training  centers  and  practicing  surgeons  can  now  use  these  data  to  identify  how  their 
skills  compare  to  laparoscopic  surgeons  nationwide,  and  to  set  performance  goals  accordingly. 


Introduction 


The  MASTER  curriculum  based  laparoscopic  cholecystectomy  (CBLC)  multicenter  trial 
is  designed  to  assess  whether  residents  trained  using  a  uniquely  designed  curriculum  that 
includes  virtual  reality  laparoscopic  training  on  the  MIST-VR  make  fewer  errors  in  the 
performance  of  laparoscopic  cholecystectomy  than  residents  who  undergo  standard 
training  without  such  a  curriculum.  This  design  requires  that  residents  randomized  to  the 
curriculum  training  arm  of  the  study  train  on  the  MIST-VR  to  a  certain  performance  level 
(or  criterion)  prior  to  performing  human  laparoscopic  cholecystectomy.  Currently,  this 
performance  level  is  not  set.  Consequently,  training  cannot  proceed  until  these  levels  are 
established. 

Based  on  previous  models  assessing  the  utility  of  the  MIST-VR  as  a  training  modality, 
this  study  calls  on  experts,  who  have  performed  at  least  100  laparoscopic 
cholecystectomies,  to  set  the  MIST-VR  performance  criterion  that  curriculum  trained 
residents  will  need  to  achieve.  The  initial  phase  of  this  trial,  therefore,  is  to  analyze  the 
MIST-VR  performance  of  experts  at  participating  institutions.  This  guide  is  intended  to 
help  your  institution  collect  the  necessary  data  so  this  analysis  can  occur  as  soon  as 
possible.  Specifically  it  outlines  the  steps  needed  to  configure  your  MIST-VR  for  trial 
participation,  and  describes  the  process  of  saving  and  packaging  the  data  for  statistical 
analysis  at  Emory. 

Before  you  set  up  your  MIST-VR,  there  are  a  few  administrative  issues  to  consider: 

1.  Your  institution  should  assign  one  MIST-VR  administrator  who  can 
configure  the  MIST-VR  software  (e.g.  Frameset),  and  can  troubleshoot 
any  software  or  hardware  complications. 

2.  Although  this  guide  is  designed  for  novice  to  intermediate  MIST-VR 
users,  your  administrator  should  have  a  working  knowledge  of  computer 
function,  with  the  ability  to  navigate  through  a  graphical  user  interface 
(e.g.  Windows)  without  difficulty. 

3.  Ideally,  your  administrator  should  provide  the  bulk  of  the  MIST-VR 
support  to  those  experts  helping  to  set  performance  levels,  as  well  as 
oversee  the  training  of  residents  randomized  to  curriculum  based  training. 

4.  The  administrator  will  need  the  authority  to  ensure  that  testing  and 
training  is  completed  in  a  timely  fashion.  This  is  particularly  important 
when  dealing  with  experts  whose  schedules  often  compete  with  the 
completion  of  the  MIST-VR  tasks  necessary  to  set  expert  performance 
levels. 

5.  In  order  to  maintain  anonymity,  your  experts  will  be  identified  using  trial 
specific  identifiers  (e.g.  Expert  1,  Expert  2,  Expert  3  etc ... ). 

We  hope  that  this  guide  will  be  helpful  in  your  endeavors.  Please  do  not  hesitate  to 
contact  us  with  any  questions  you  may  have. 
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Section  1:  Preliminary  Startup 

To  conduct  this  trail  successfully,  it  is  important  to  recruit  as  many  laparoscopic  experts 
as  possible.  In  this  way,  we  will  be  able  to  ensure  that  the  performance  criterion  levels 
that  are  set  for  the  residents  undergoing  curriculum  based  training  accurately  reflect  a 
relatively  normal  distribution  of  expertise  seen  throughout  participating  centers.  As  such, 
we  expect  a  number  of  experts  to  participate  from  each  center.  Currently  at  Emory, 
seven  experts  have  volunteered. 

Officially,  we  are  defining  an  “expert”  as  any  individual  that  has  performed  100  or  more 
laparoscopic  cholecystectomies.  Once  identified,  an  expert  will  perform  5  trials  of  the  6 
MIST-VR  Core  Skills  1  tasks  of  AcquirePlace,  TransferPlace,  Traversal,  Withdrawlnsert, 
Diathermy,  and  Manipulation  Diathermy  using  the  specific  task  configurations  set  for  the 
MASTER  CBLC  multicenter  trial. 

Each  individual  trial  (of  all  6  tasks)  should  be  completed  in  its  entirety  during  a  given 
session.  Experts  may  complete  several  complete  trials  in  one  session  if  they  so  choose. 
We  recommend  that  they  not  complete  all  five  trials  in  a  single  session,  however.  On 
average,  each  trial  takes  approximately  forty-five  minutes  to  complete.  The  unique 
CBLC  task  settings  are  purposefully  difficult  and  will  take  longer  to  complete  than  when 
performed  on  the  default  easy,  medium,  or  even  hard,  settings  that  your  experts  may  be 
accustomed  to. 
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If  an  expert  is  completely  unfamiliar  with  the  MIST-VR  environment,  a  practice  run  on 
the  medium  MIST-VR  task  settings  can  be  performed.  In  this  case,  a  detailed 
explanation  of  the  MIST-VR  scoring  matrix  (especially  the  errors  scored)  should  be 
undertaken  prior  to  the  first  performance  of  each  task,  unless  the  expert  is  familiar  with 
the  MIST-VR  environment.  If  you  are  unfamiliar  with  these  particular  metrics,  please 
refer  to  the  MIST-VR  documentation  provided  with  you  simulator. 

Each  expert  will  have  their  own  preference  as  to  how  they  would  like  their  environment 
set.  Often  darkened  lighting,  a  quiet  room,  or  adjustment  of  the  instruments  will  be 
requested.  To  accommodate  such  requests,  we  have  not  set  a  standard  testing 
environment.  Unusual  requests  (e.g.  sitting)  will  need  to  be  discussed  with  the  trial 
coordinators  prior  to  acceptance,  however. 

Lastly,  there  are  two  points  about  the  scoring  metrics  that  should  be  discussed  with 
experts  prior  to  starting  these  tasks.  First,  we  are  primarily  concerned  with  errors  and 
instrument  economy.  In  this  respect,  time  to  completion  is  weighed  less  heavily  in  our 
calculations.  Within  the  MIST-VR,  however,  time  is  a  significant  factor  affecting  the 
final  score.  For  this  reason,  final  scores  should  not  be  provided  as  a  gauge  for  expert 
performance  progression.  This  is  also  why  using  the  graphing  function  provided  by 
FrameSET  (e.g.  peer-to-peer  or  progression)  are  not  used  during  these  expert  trials. 
Second,  during  the  last  task  (manipulation  diathermy)  it  is  okay  to  pull  the  sphere  out  of 
the  large  box  (after  initially  grasping  it)  in  order  to  position  the  diathermy  box  into  a 
safer,  more  acceptable  position.  The  MIST-VR  environment  will  sometimes  place  the 
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diathermy  box  in  an  unsafe  position  (e.g.  completely  behind  the  sphere,  or  very  close  to 
the  grasping  instrument).  In  these  cases  we  prefer  to  score  safety  over  exceptional 
dexterity,  and  therefore  have  eliminated  this  particular  error  from  the  manipulation 
diathermy  task.  The  precedent  behind  both  of  these  adjustments  was  set  during  previous 
trials  studying  the  effectiveness  of  the  MIST-VR  as  a  means  to  train  safer  surgeons. 
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Section  2:  Configuring  MASTER  CBLC  specific  settings 


Step  1:  Load  FrameSET  by  double  clicking  the  Frameset  icon  shown 
In  Figure  1.  If  you  cannot  find  this  icon  readily,  the  Frameset  program 
can  be  found  in  the  windows  folder:  (c:/Program  Files/FrameSET/). 


Step  2:  Login  to  the  FrameSET  administrator  mode  by  using  your  administrative 
username  and  password.  If  you  do  not  remember  these  codes,  remember  that  Mentice 
occasionally  sets  the  administrator  default  to  username  =  a;  password  =  a. 


Step  3:  Once  inside  the  FrameSET  administrator  mode, 
left  click  on  the  plus  sign  to  the  left  of  the  FrameSET  menu 
choice  in  the  left  sided  window  pane  (Figure  2).  Several 
branches  will  present  under  the  FrameSET  data  tree.  In 
this  section  you  will  be  concerned  with  the  Tasks  option. 

Step  4:  Left  click  on  the  plus  sign  to  the  left  of  the  Tasks 
folder  as  shown  in  Figure  3.  The  branches  off  the  Tasks 
folder  are  all  the  MIST-VR  tasks  available  on  your 
computer.  Within  the  MASTER  CBLC  trial,  we  are  using 
the  Core  Skills  1  task  set.  This  includes  the  following 
tasks:  Acquire  Place,  Transfer  Place,  Traversal, 

Withdraw/Insert,  Diathermy,  and  Manipulation  Diathermy.  For  now,  you  will  set  a  new 
configuration  for  Acquire  Place. 


'  FrameSET 


£ile  :  ipew:  Options  Wizards  Users 


S  i  I>i  J  . 

Admin  User  Class  J 


[•  Users 
1  "1  Administrators 

Tasks 
S  etups 

gj  D  ata  Analysis 
Content 
Extras 


Figure  3 


FrameSET 
Figure  1 
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Step  5:  Left  click  on  the  plus  sign  to  the  left  of  the  Acquire  Place  choice  (Figure  4). 

This  will  branch  into  Configs  and  Weightings.  Right  click  on  the  Configs  text  (not  on  the 
arrow),  and  choose  Add  Config  from  the  pop-up  menu  item.  In  the  right  window  pane,  a 


new  window  will  appear.  You  screen  should  look  like  Figure  4. 


0e  View  Options  Wizards  Users  Administrators  Tasks  Setups  Oats  Analysis  Content  ;  He£ 

i . S  i  1‘ii it . FT If . I* i 


Admin  User 
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Setup  Config^  Weight  Graph  Content  Assign  Install  Remove 
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Figure  4 


Step  6:  Change  the  name  of  the  New  Configuration  (circled  in  Figure  4),  to  MASTER 
CBLC.  Once  completed,  click  apply  (circled  in  Figure  5). 
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Step  7:  Direct  your  attention  to  the  circled  area  of 
Figure  6  (above).  Here  is  where  the  unique 
MASTER  CBLC  configuration  settings  will  be 
applied.  Change  these  settings  to  the  following 
specifications:  Target  Sphere  =  7.5;  Targ.  =  9.25 
(you  will  have  to  highlight  the  number  and  type  it  in 
yourself.  The  arrows  will  not  allow  you  to  go  below 
10);  Camera  Pos  =  100.  Once  complete,  click  on 
apply,  and  check  to  make  sure  your  settings  match 
those  in  Figure  7.  When  complete,  close  the  configuration  window. 

Step  8:  The  configuration  for  the  Acquire  Place  task  is  now  complete.  Take  similar 
steps  to  configure  the  remaining  5  tasks  using  the  settings  listed  in  Appendix  1.  Be  sure 
to  close  each  configuration  window  when  you  are  done  changing  the  individual  tasks  - 
otherwise  FrameSET  will  display  an  error  window. 
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Section  3:  Adding  Classes  and  Users  to  MIST-VR 


Step  1 :  Left  click  the  plus  sign  to  the  left  of  the  Users  option  within  the  FrameSET  data 
tree  (Figure  1).  This  will  display  the  user  classes  (or  groups) 
currently  installed  within  your  copy  of  FrameSET.  In  this 
section  you  will  be  creating  your  Expert  class. 


Step  2:  The  easiest  way  to  create  a  class  is  to  left  click  on  the 
Class  icon  located  on  the  FrameSET  menu  bar  at  the  top  of  the 
main  window  (Figure  2).  Next,  type  MASTER  CBLC  Expert 
for  Class  Name  at  the  prompt  (Figure  3). 


Figure  1 


When  you  are  finished  typing,  left  click  on  the  next  button  at  the  bottom  of  the  window. 
The  window  will  then  ask  whether  you  want  to  add  users  to  this  class.  We  have  found 
that  adding  users  in  this  fashion  will  crash  the  program;  therefore  you  do  not  want  to 
create  users  at  this  time.  Continue  on  by  clicking  the  Next  button  again.  Your  class 
should  appear  under  the  Users  data  tree  in  the  window  to  the  left. 
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Step  3:  To  add  users,  you  can  either  a)  left 
click  on  the  Users  icon  on  the  FrameSET 
menu  bar  (Figure  4);  or  b)  right  click  on  the 
MASTER  Expert  Class  item  found  in  the  data 
tree  to  the  left. 


a)  If  you  use  the  Users  icon,  use  the  subject’s  number  as  the  account  name,  and  be  sure 

to  left  click  on  the  Enter  Now  radial  button  in 
order  to  add  more  details  (Figure  5).  Left  click 
the  Next  button  on  the  bottom  of  the  window  to 
continue.  Within  the  details  section,  use  the 
expert’s  number  as  his/her  last  name  (e.g.  Expert 
1).  Next,  choose  a  password  that  you  can 
remember.  Confirm  the  password  and  click  the 
Next  button.  The  next  window  will  ask  you  to  select  a  user  class  for  this  new  user.  For 
this  example  you  will  choose  the  MASTER  Expert  class. 


b)  If  you  right  click  on  the  MASTER  Expert  Class  in 
the  data  tree,  add  the  subject’s  number  as  the  account 
name  and  user  first  name.  Next,  choose  a  password 
you  will  remember  (Figure  6).  Click  apply  when 
you  are  finished. 
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Figure  6 
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Section  4:  Creating  a  MIST-VR  task  setup  with  assignments 


Once  you  have  designed  a  class  and  placed  experts  into  it,  you  will  need  to  add  a  task 
setup  to  the  group.  The  task  setup  is  the  sequence  of  MIST-VR  tasks  that  each  user  will 
be  asked  to  perform.  Since  you  are  using  the  same  task  configuration  settings  that  will  be 
used  when  residents  are  training,  you  can  use  the  same  setup  for  both  expert  and  training 
classes  (once  that  class  is  made  later  in  the  trial).  Here  you  create  the  MASTER 
Training  setup  that  will  be  used  for  both  of  these  groups  -  although  it  is  specifically  used 
for  the  experts  in  this  initial  portion  of  the  trial. 

Step  1 :  Left  click  on  the  Setup  icon  located  on  the  FrameSET  menu  bar  at  the  top  of  the 

main  window  (Figure  1).  Type  MASTER  Training 
in  the  setup  name  box.  Move  on  to  the  next  step  by 
left  clicking  on  the  finish  button. 


Step  2:  You  will  need  to  add  content,  as 
introductory  segments  to  the  various  skill  tasks,  as 
well  as  the  tasks  themselves.  Add  the  introduction 
to  the  acquire  place  MIST-VR  task  by  left  clicking 
on  the  add  button  under  the  components  portion  of 
the  window  (Figure  3).  When  the  pop-up  menu 
appears,  left  click  on  the  Content  option. 


;#e  yjevt  Options  Wizards  Users  Administrators  Tasks  Set 

. i  s . i5ij(S) i; i 

idmfn  User  Class  ^Setup^/  Ccnfig  Weight 
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Figure  1 
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Next,  left  click  the  arrow  to  the  left  of  the 
Expert  Demo  option.  Choose  the  AP  Demo 
option  and  check  to  make  sure  it  was  added  to 
your  setup  listing  (Figure  4). 


Step  3:  Now  add  the  Acquire  Place  task.  Left 
click  on  the  add  button  under  the  components 
portion  of  the  window.  When  the  pop-up 
appears,  choose  the  Task  option.  Choose 
AcquirePlace  and  then  left  click  the  Next 
button.  A  new  window  displaying  the  various 
task  configuration  options  now  appears. 
Choose  the  Medium  configuration  and  left  click  on  the  Finish  button. 


Step  4:  Continue  adding  content  and  MASTER 
CBLC  configuration  tasks  in  the  following  order: 
Content  ->  TP  demo;  Task  ->  TransferPlace; 
Content  ->  TV  demo;  Task  ->  Traversal;  Content 
->  WI  Insert;  Task  ->  Withdrawlnsert;  Content  -> 
DT  demo;  Task  ->  Diathermy;  Content  ->  MD 
demo;  Task  Manipulation  Diathermy.  When 
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you  are  finished  your  window  should  look  like  Figure  5. 


Step  5:  Each  expert  will  need  to  be  assigned  a  task  setup 
before  using  the  MIST-VR  system.  Assignment  is 
accomplished  by  left  clicking  on  a  setup  and  dragging  the 
setup  to  a  given  user  name.  If  the  users  of  a  given  class  are 
going  to  perform  the  same  tasks,  you  can  save  time  by 
dragging  the  setup  over  the  class  name  instead  of  each 
individual  user  (Figure  6).  For  the  MASTER  lap.  chole. 
trial  the  Training  setup  will  be  applied  to  the  MASTER 
Expert  class. 
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Section  5:  Exporting  MIST-VR/FrameSET  data  to  Excel 


Three  sets  of  data  will  be  needed  for  completion  of  this  MASTER  trial:  the  expert 
performance  levels  for  each  task,  the  results  of  the  preliminary  psychomotor  testing  of  the 
individual  subjects,  and  the  performance  of  each  subject  during  training.  We  have 
chosen  to  accept  MIST-VR  data  for  analysis  in  the  Excel  spreadsheet  format. 
Unfortunately,  there  is  no  easy  way  to  export  all  MIST-VR  data  from  a  specific  user  class 
into  the  same  Excel  spreadsheet.  Instead,  FrameSET  only  allows  task  specific  data  to 
exist  on  the  same  spreadsheet.  For  example,  let’s  assume  you  have  two  experts  at  your 
institution  who  have  completed  their  5  MIST-VR  trials  to  help  set  the  expert  criterion  for 
the  study.  In  order  to  have  their  data  processed  for  statistical  analysis,  you  will  save  six 
different  Excel  files  -  one  for  each  individual  MIST-VR  task.  In  the  case  of  the 
AcquirePlace  task,  this  is  conceptually  depicted  in  Figure  1. 


Expert  1  Expert  2 


Figure  1 
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In  the  same  example,  if  you  enroll  eight  subjects  with  four  of  them  randomizing  to  VR 

training,  you  will  eventually 
send  a  total  of  36  Excel  files 
to  Emory.  This  is  shown  in 
Figure  2.  Also  note  that  this 
figure  has  the  number  of 
rows  each  Excel  file  should 
contain  (e.g.  2  experts,  5 
trials  =10  rows).  There  are 
question  marks  underneath 
the  tasks  performed  by  the 
trainees  due  to  our  inability 
to  predict  how  many  training  trials  it  will  take  to  reach  performance  criterion  levels. 

Now,  with  an  understanding  of  what  you  will  be  sending,  what  follows  is  a  description  of 
the  export  and  saving  process  using  the  psychomotor  testing  data  as  an  example.  Like 
previous  examples,  mimicking  these  steps  will  allow  you  to  save  and  package  the 
remaining  data  when  needed.  If  you  do  not  have  Excel 
installed  on  the  computers  running  FrameSET,  there  is  an 
alternative  work-around  that  is  described  in  Appendix  3. 


Step  1 :  Navigate  through  the  Frameset  data  tree  (left 
clicking  the  plus  signs  to  the  left  of  the  text  items)  to  find  the 
users  in  the  MASTER  Expert  class.  Left  click  the  plus  sign 
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on  each  of  the  users  until  you  see  a  Results  item  under  each  user  (Figure  3). 

Step  2:  Left  click  on  the  plus  sign  next  to  the  Results  item  under  the  first  user  listed  (a 
plus  sign  will  appear  after  the  expert  has  completed  a  trial).  The  MIST-VR  Core  Skills  1 
tasks  are  then  listed  beneath.  Left-click  the  plus  sign  next  to  the  AcquirePlace  task. 

Step  3:  Underneath  you  will  find  a  Cf  (or  configurations)  folder  named  Medium.  Within 
this  folder  are  the  raw  scores  of  the  user’s  one  trial  of  AcquirePlace  during  the  initial 
psychomotor  testing.  Left  click  on  the  plus  sign  to  see  the  Re  (or  Results)  folder  named 
with  the  date  and  time  the  task  was  performed. 

Step  4:  Right  click  on  the  results  folder  and  choose  the  Display  option  from  the  pop-up 
menu.  You  should  see  a  window  labeled  “Display  Results  -  AcquirePlace,  Medium 
Configuration”  in  the  right  sided  window  pane  (Figure  4). 


Step  5:  You  will  now  need  to  add  the  remainder  of  the  experts’  AcquirePlace  results  in 
the  display  results  window  by  repeating  Steps  2-4  for  each  user. 
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Step  6:  Export  the  data  to  an  Excel  spreadsheet  by  left  clicking  on  the  Export  button  on 
the  bottom  of  the  display  results  window  (Figure  5).  The  program  will  then  ask  for  a  file 
name.  As  listed  in  Appendix  2,  the  file  name  for  this  file  (collected  at  Emory)  should  be 
MastLCEmoryExpertAP .  Choose  a  folder  to  save  the  Excel  file  in,  and  left  click  on  the 
Save  button  at  the  bottom  of  the  window. 


Step  7:  Repeat  Steps  2-6  for  each  respective  MIST-VR  task.  Use  the  naming 
conventions  listed  in  Appendix  2. 


Step  8:  Place  all  Excel  files  on  a  CD-Rom,  or  3.5”  floppy  disk  and  send  them  to  the 


Emory  Endosurgery  Center  as  listed  on  the  title  page. 
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Appendix  Is  MIST-VR  CBLC  configurations 


Target 

Sphere 

Target 

Box 

Diathermy 

Box 

Length 

Radius 

Number 

Camera 

Position 

Acquire  Place 

7.5 

9.25 

100 

Transfer  Place 

7.5 

9.25 

100 

Traversal 

I 

i 

I 

i 
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i 

i 

5 

2.5 

6 

100 

W  ithdraw/Insert 

7 

9 

100 

Diathermy 

4 

3.5 
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Manipulation 
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7.5 
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Appendix  2:  File  naming 


File  naming  template:  MastLC  YFZ.xls 

MastLC  stands  for  MASTER  Laparoscopic  Cholecystectomy 
X  =  Institution  name 
Y  =  Study  component 
Z  =  MIST-VR  task 


_ Institution  name  abbreviations _ • _ 

Use  the  first  five  letters  of  your  institution  name.  Alternatively,  you  can  choose  an 
abbreviation  <  5  characters  that  may  be  more  fitting  as  an  identifier  (e.g.  UNC). 


Study  component  abbreviations 

Setting  expert  criterion 

Expert 

Psych 

Training 

Train 

MIST-VR  task  abbreviations 

AcquirePlace 

AP 

TransferPlace 

tp  1 

Traversal 

TV 

Withdrawlnsert 

WI  j 

Diathermy 

DT  I 

Manipulation  Diathermy 

MD 

Example:  For  a  group  of  Emory  University  experts  who  have  finished  their  5  MIST-VR 
trials,  the  name  for  the  Manipulation  Diathermy  task  results  file  should  be: 

MastLCEmoryExpertMD 
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Appendix  3:  Exporting  data  without  using  Excel 


This  appendix  is  intended  for  institutions  that  are  unable  to  use  Excel  as  their  means  of 
data  presentation  and  packaging  for  analysis.  To  see  how  to  manipulate  FrameSET  in 
order  to  export  user  data  for  the  MASTER  Laparoscopic  Cholecystectomy  trial,  please 
refer  to  Section  4  in  this  instruction  guide. 

To  export  your  data  you  will  need  to  have  a  text  editor  loaded  on  the  computer  the  data  is 
being  exported  from.  Examples  include  the  Microsoft  programs  NotePad,  or  WordPad 
that  are  provided  with  the  Microsoft  operating  system  software.  Most  likely,  one  of  these 
programs  is  already  loaded  on  your  computer,  and  can  be  found  within  the  Accessories 
folder  located  within  the  programs  region  of  the  Start  menu. 

In  Section  4  of  the  instruction  guide,  Step  6  asks  you  to  click  on  an  export  button  that 
automatically  saves  data  in  an  Excel  format.  Instead  of  performing  this  task,  you  will 
need  to  cut  and  paste  all  of  the  user  data  from  the  results  window  into  your  text  editor. 

Step  1 :  Using  your  mouse,  highlight  all  the  users  and  their  results. 

Step  2:  Making  sure  that  all  the  data  remains  highlighted,  click  on  the  Copy  button  on 
the  lower  portion  of  the  window.  This  will  place  all  the  highlighted  data  in  the  windows 
clipboard  so  that  you  can  paste  it  into  your  text  editing  program. 

Step  3:  Paste  the  data  into  your  text  editing  program  using  the  appropriate  commands  for 
the  program.  In  NotePad  or  WordPad,  you  can  find  the  paste  command  under  the  Edit 
menu  item  on  the  top  toolbar. 

Step  4:  Save  the  data  using  the  appropriate  file  name  as  discussed  in  Appendix  2.  Files 
saved  using  this  method  will  have  a  .txt  extension  instead  of  the  .xls  extension  discussed 
above. 
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